Method and electronic device for auto focus of scene

ABSTRACT

An electronic device for auto focus is provided. The electronic device includes determining, by the electronic device, at least one region of interest (ROI) in a scene displayed in one of a viewfinder and a captured image frame and determining, by the electronic device, at least one sub ROI in the at least one ROI by performing a first level of auto focus using at least one first image sensor. Further, the method includes determining, by the electronic device, at least one focused sub ROI by performing a second level of auto focus on the at least one sub ROI using at least one second image sensor and rendering, by the electronic device, a focus transition for the at least one focused sub ROI to one of the viewfinder and the captured image frame.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is an International Application No. PCT/KR2022/007055designating the United States, filed on May 17, 2022, in the KoreanIntellectual Property Receiving Office and claiming priority to IndianPatent Application No. 202141038095, filed on Aug. 23, 2021, in theIndian Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND Field

The disclosure relates to image and video processing. More particularly,the disclosure relates to a method and an electronic device for fasterand enhanced auto focus of a scene displayed in a viewfinder or from acaptured image frame.

Description of Related Art

Dual Pixel autofocus is an increasingly popular smartphone camerafeature, particularly at the flagship end of the market. The technologypromises much faster focusing for action shots as well as superiorfocusing in low-light environments. Dual Pixel autofocus is an extensionof Phase Detection autofocus (PDAF), featured in smartphone cameras forsome years. Essentially, PDAF uses dedicated left-looking andright-looking pixels on the image sensor to calculate whether the imageis in focus.

PDAF is the precursor to Dual Pixel autofocus, so understanding how theformer works is essential. PDAF is based on the slightly differentimages created from masked “left-looking and right-looking” photodiodesembedded into the image sensor's pixels. Comparing the phase differencebetween these pixels is used to calculate the required focus distance.These phase-detection pixels typically make up just around 5-10% of allthe image sensor's pixels. Using a larger number of dedicatedphase-detection pixel pairs makes PDAF more reliable and accurate. Inthe move to Dual Pixel AF, every pixel on the sensor is used for PDAFand aids in calculating phase differences and focus. This improvesaccuracy and speed compared to standard PDAF. Each pixel is split intotwo photodiodes; one left and right looking. Using micro-lenses placedon-top pixels makes this possible. When taking a photo, the processoranalyzes the focus data from each photodiode first before combining thesignals to record the full pixel used in the final image.

However, dual pixel technology uses higher processing power. Higherprocessing power used for all pixels hampers performance and slows downother functions as well as limits value addition to camera. Due tohigher processing power for calculation of auto focus using all dualpixels, many advanced optimizations and effects cannot be applied forenhancing the user experience or it may lead to lag or furtherdeterioration of battery. Further, edge objects and objects that are ona depth axis are not easily captured using a single camera. Focusingoperations do not utilize information directed to semantic relationshipsbetween regions of interest in an image.

Thus, it is desired to at least provide a mechanism for auto focus thatis devoid of the above issues.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea method of intelligently activate/deactivate dual pixels in order toachieve a fast, reliable and resource effective auto-focus mechanismwhile having optimization using multiple cameras.

Another aspect of the disclosure is to determine an importance hierarchyto focus on parts which are deemed to be of more importance.

Another aspect of the disclosure is to determine the pixel densities ofdual pixels intelligently for calculation of focus according to the ROIsdetected in image, user preferences.

Another aspect of the disclosure is to select only strategic dual pixelswhose left and right signals are used for focus calculation based onIntelligence criteria and multiple cameras.

Another aspect of the disclosure is to detect at least one region ofinterest (ROI) in a scene displayed in a viewfinder or from a capturedimage frame.

Another aspect of the disclosure is to allocate the at least one ROI toat least one first image sensor from the plurality of image sensors toperform a first level of auto focus.

Another aspect of the disclosure is to perform the first level of autofocus on the at least one ROI using the at least one first image sensorto obtain at least one sub ROI of the at least one ROI.

Another aspect of the disclosure is to allocate the at least one sub ROIto at least one second image sensor from the plurality of image sensorsto perform a second level of auto focus.

Another aspect of the disclosure is to perform the second level of autofocus on the at least one sub ROI using the at least one second imagesensor to obtain at least one focused sub ROI.

Another aspect of the disclosure is to merge focused sub-ROIs and torender a focus transition for the at least one focused sub ROI to theviewfinder or the captured image frame.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, a method for auto focusof a scene by an electronic device is provided. The method includesdetermining, by the electronic device, at least one region of interest(ROI) in the scene displayed in one of a viewfinder and a captured imageframe and determining, by the electronic device, at least one sub ROI inthe at least one ROI by performing a first level of auto focus on the atleast one ROI using at least one first image sensor of the plurality ofimage sensors. Further, the method includes determining, by theelectronic device, at least one focused sub ROI by performing a secondlevel of auto focus on the at least one sub ROI using at least onesecond image sensor of the plurality of image sensors; and rendering, bythe electronic device, a focus transition for the at least one focusedsub ROI to one of the viewfinder and the captured image frame.

In an embodiment, determining, by the electronic device, the at leastone sub ROI in the at least one ROI by performing the first level ofauto focus on the at least one ROI using the at least one first imagesensor of the plurality of image sensors includes determining, by theelectronic device, a relative importance factor for each of the at leastone ROI and allocating, by the electronic device, the at least one ROIto the at least one first image sensor based on the relative importancefactor and a plurality of parameters. Further, the method includesselecting, by the electronic device, dual pixels using a pixel densityfunction to focus on of the at least one ROI for focus calculation andactivating, by the electronic device, a set of two photo-diodes of theat least one first image sensor corresponding to the selected dualpixels for performing the first level of auto focus. The method alsoincludes performing, by the electronic device, the first of auto focuscorresponding to the selected dual pixels of the at least one ROI usingan activated diode; and determining, by the electronic device, the atleast one sub ROI in the at least one ROI based on the first level ofauto focus performed on the at least one ROI.

In an embodiment, the selected dual pixels are located at boundary ofthe at least one ROI.

In an embodiment, determining, by the electronic device, the relativeimportance factor for each of the at least one ROI includes determining,by the electronic device, a semantic relevance of the at least one ROIto the scene and determining, by the electronic device, at least oneobject in the at least one ROI and a user preference with respect to thedetermined at least one object. Further, the method includesconcatenating, the semantic relevance of the at least one ROI, the atleast one object in the at least one ROI and the user preference withrespect to the determined at least one object and determining, by theelectronic device, the relative importance factor for each of the atleast one ROI based on the concatenation.

In an embodiment, allocating, by the electronic device, the at least oneROI to the at least one first image sensor based on the relativeimportance factor and the plurality of parameters includes determining,by the electronic device, the plurality of parameters associated withthe at least one ROI, wherein the plurality of parameters comprises afocal length of an image sensor, a type of the at least one ROI, theimportance of the at least one ROI, a line of sight of each image sensorof the electronic device, and a resolution of each image sensor of theelectronic device. Further, the method also includes selecting, by theelectronic device, the at least one first image sensor from theplurality of image sensors to perform the first level of auto focusbased on the plurality of parameters and allocating, by the electronicdevice, the at least one ROI to the at least one first image sensor fromthe plurality of image sensors to perform the first level of auto focus.

In an embodiment, determining, by the electronic device, the at leastone focused sub ROI by performing the second level of auto focus on theat least one sub ROI using the at least one second image sensor of theplurality of image sensors includes allocating, by the electronicdevice, the at least one sub ROI to at least one second image sensorfrom the plurality of image sensors to perform the second level of autofocus and selecting, by the electronic device, dual pixels using a pixeldensity function to focus on the at least one sub ROI for focuscalculation. Further, the method also includes activating, by theelectronic device, a set of two photo-diodes of the at least one secondimage sensor corresponding to the selected dual pixels for performingthe second level of auto focus and performing, by the electronic device,the second level of auto focus corresponding to the selected dual pixelsof the at least one sub ROI using at least one activated pixel to obtainthe at least one focused sub-ROI

In an embodiment, the pixel density function is determined as activepixel density function for the at least one focused sub-ROI as a mixtureof Gaussians and wherein with each Gaussian is centered at a landmark.

In an embodiment, allocating, by the electronic device, the at least onesub ROI to at least one second image sensor from the plurality of imagesensors to perform the second level of auto focus includes segregating,by the electronic device, the at least one ROI into a plurality ofsub-ROIs and detecting, by the electronic device, at least one landmarkon each of the plurality of sub-ROIs. The method also includesestimating, by the electronic device, an importance factor for eachsub-ROI based on the detected at least one landmark on each of theplurality of sub-ROIs and selecting, by the electronic device, the atleast one sub-ROI from the plurality of sub-ROIs based on the importancefactor and the detected landmarks. Further, the method also includesselecting, by the electronic device, the at least one second imagesensor supporting dual pixel density function from the plurality ofimage sensors; and allocating, by the electronic device, the at leastone sub-ROI to the at least one selected second image sensor forperforming the second level of auto focus.

In an embodiment, estimating, by the electronic device, the importancefactor for each sub-ROI based on the detected at least one landmark oneach of the plurality of sub-ROIs includes determining, by theelectronic device, a number of ROIs detected in the sub-ROI. The methodalso includes determining, by the electronic device, a user preferencecorresponding to the sub-ROI and estimating, by the electronic device,the importance factor for the sub-ROI using the number of ROIs detectedin the sub-ROI and the user preference corresponding to the sub-ROI.

In an embodiment, rendering, by the electronic device, the focustransition for the at least one focused sub ROI to one of the viewfinderand the captured image frame includes merging, by the electronic device,the at least one focused sub-ROI obtained from the at least one secondimage sensor to obtain an optimal sub-ROI. The method also includesdetermining, by the electronic device, the focus transition fordisplaying the optimal sub-ROI; and rendering, by the electronic device,the focus transition for the optimal sub-ROI to one of the viewfinderand the captured image.

In an embodiment, the focus transition is determined based on at leastone of user preference and a hierarchy important of the at least oneROI, a type of the at least one ROI, and an importance of the at leastone ROI.

In accordance with another aspect of the disclosure, an electronicdevice for auto focus on a scene is provided. The electronic deviceincludes a memory, a processor, a communicator, a plurality of imagesensors and a focus optimization controller. The focus optimizationcontroller is configured to determine at least one region of interest(ROI) in the scene displayed in one of a viewfinder and a captured imageframe and determine at least one sub ROI in the at least one ROI byperforming a first level of auto focus on the at least one ROI using atleast one first image sensor of the plurality of image sensors. Thefocus optimization controller is also configured to determine at leastone focused sub ROI by performing a second level of auto focus on the atleast one sub ROI using at least one second image sensor of theplurality of image sensors, and render a focus transition for the atleast one focused sub ROI to one of the viewfinder and the capturedimage frame.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured, when determining therelative importance factor for each of the at least one ROI, to:determine a semantic relevance of the at least one ROI to the scene;determine at least one object in the at least one ROI and a userpreference with respect to the determined at least one object;concatenate the semantic relevance of the at least one ROI, the at leastone object in the at least one ROI and the user preference with respectto the determined at least one object; and determine the relativeimportance factor for each of the at least one ROI based on theconcatenation.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured, when determining therelative importance factor for each of the at least one ROI, to:determine a semantic relevance of the at least one ROI to the scene;determine at least one object in the at least one ROI and a userpreference with respect to the determined at least one object;concatenate the semantic relevance of the at least one ROI, the at leastone object in the at least one ROI and the user preference with respectto the determined at least one object; and determine the relativeimportance factor for each of the at least one ROI based on theconcatenation.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured, when allocating the atleast one ROI to the at least one first image sensor based on therelative importance factor and the plurality of parameters, to:determine the plurality of parameters associated with the at least oneROI, wherein the plurality of parameters comprises a focal length of animage sensor, a type of the at least one ROI, the importance of the atleast one ROI, a line of sight of each image sensor of the electronicdevice, and a resolution of each image sensor of the electronic device;select the at least one first image sensor from the plurality of imagesensors to perform the first level of auto focus based on the pluralityof parameters; and allocate the at least one ROI to the at least onefirst image sensor from the plurality of image sensors to perform thefirst level of auto focus.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured, when determining the atleast one focused sub-ROI by performing the second level of auto focuson the at least one sub-ROI using the at least one second image sensorof the plurality of image sensors, to: allocate the at least one sub-ROIto at least one second image sensor from the plurality of image sensorsto perform the second level of auto focus; select dual pixels using apixel density function to focus on the at least one sub-ROI for focuscalculation; activate a set of two photo-diodes of the at least onesecond image sensor corresponding to the selected dual pixels forperforming the second level of auto focus; and perform the second levelof auto focus corresponding to the selected dual pixels of the at leastone sub-ROI using at least one activated pixel to obtain the at leastone focused sub-ROI.

In an embodiment, the above electronic device, wherein the pixel densityfunction is determined as an active pixel density function for the atleast one focused sub-ROI as a mixture of Gaussians, and wherein witheach Gaussian is centered at a landmark.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured, when allocating the atleast one sub-ROI to at least one second image sensor from the pluralityof image sensors to perform the second level of auto focus, to:segregate the at least one ROI into a plurality of sub-ROIs; detect atleast one landmark on each of the plurality of sub-ROIs; estimate animportance factor for each sub-ROI based on the detected at least onelandmark on each of the plurality of sub-ROIs; select the at least onesub-ROI from the plurality of sub-ROIs based on the importance factorand the detected at least one landmark; select the at least one secondimage sensor supporting dual pixel density function from the pluralityof image sensors; and allocate the at least one sub-ROI to the at leastone selected second image sensor for performing the second level of autofocus.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured, when estimating theimportance factor for each sub-ROI based on the detected at least onelandmark on each of the plurality of sub-ROIs, to: determine a number ofROIs detected in the sub-ROI; determine a user preference correspondingto the sub-ROI; and estimate the importance factor for the sub-ROI usingthe number of ROIs detected in the sub-ROI and the user preferencecorresponding to the sub-ROI.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured, when rendering the focustransition for the at least one focused sub-ROI to one of the viewfinderand the captured image frame, to: merge the at least one focused sub-ROIobtained from the at least one second image sensor to obtain an optimalsub-ROI; determine the focus transition for displaying the optimalsub-ROI; and render the focus transition for the optimal sub-ROI to oneof the viewfinder and the captured image.

In an embodiment, the above electronic device, wherein the focustransition is determined based on at least one of a user preference anda hierarchy important of the at least one ROI, a type of the at leastone ROI, or an importance of the at least one ROI.

In an embodiment, the above electronic device, wherein the importancefactor of each of the ROI is based on user preferences, a globalsemantic relevance of each ROI to whole image, and a general preference.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured to assign an image sensorof the plurality of image sensors to each ROI based on properties of theimage sensor, an ROI importance, and a line of sight.

In an embodiment, the above electronic device, wherein the focusoptimization controller is further configured to select pixels on aboundary of an ROI based on the pixel density function.

According to another aspect of one or more embodiments, there isprovided a computer-readable storage medium, having a computer programstored thereon that performs, when executed by a processor, the abovemethod.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates traditional phase detection autofocus (PDAF),according to the related art;

FIG. 2 illustrates optimized dual pixel autofocus (DPAF), according toan embodiment of the disclosure;

FIG. 3 illustrates a block diagram of an electronic device for autofocus on a scene, according to an embodiment of the disclosure;

FIG. 4A illustrates various functions performed by a ROI detection andfocus approximation controller, according to an embodiment of thedisclosure;

FIG. 4B illustrates a method for determination of ROI importance by aROI importance determining unit, according to an embodiment of thedisclosure;

FIG. 4C illustrates functions of a camera allocation unit, according toan embodiment of the disclosure;

FIG. 4D illustrates functions of a boundary detection and pixelactivating unit, according to an embodiment of the disclosure;

FIG. 4E illustrates a method for controlling dual pixels array based ona static pixel usage table, according to the related art;

FIG. 4F illustrates a method for dynamically controlling a dual pixelsarray, according to an embodiment of the disclosure;

FIG. 5A illustrates various functions performed by a sub-ROI segregationand image sensor assignment controller, according to an embodiment ofthe disclosure;

FIG. 5B illustrates functioning of the ROI segregation unit, accordingto an embodiment of the disclosure;

FIG. 5C illustrates functioning of a pixel density allocating unit,according to an embodiment of the disclosure;

FIG. 6A illustrates various functions performed by a presentationmanagement controller, according to an embodiment of the disclosure;

FIG. 6B illustrates various functions performed by a transition effectdetermining unit, according to an embodiment of the disclosure;

FIG. 7A, 7B, 7C, 7D, 7E, and 7F are flow diagrams illustrating asequence of operations for two levels of auto focus with respect to anelectronic device, according to various embodiments of the disclosure;

FIGS. 8A and 8B are examples illustrating multiple ROIs with different Zorders being captured by an electronic device, according to anembodiment of the disclosure;

FIG. 9 is an example illustrating a comparison of capturing of contentin an edge of images by an electronic device, according to an embodimentof the disclosure;

FIG. 10 is an example illustrating capture of a sudden intruding objectby an electronic device, according to an embodiment of the disclosure;

FIG. 11A illustrates a comparison of existing method and proposed methodof focusing on the objects on viewfinder, according to an embodiment ofthe disclosure;

FIG. 11B is an example illustrating an operation-by-operation procedurefor providing real-time suggestions for the objects to be focused on theviewfinder, according to an embodiment of the disclosure; and

FIG. 12 is an example illustrating a scenario of threat identificationin the scene by the electronic device, according to an embodiment of thedisclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

The embodiments herein and the various features and advantageous detailsthereof are explained more fully with reference to the non-limitingembodiments that are illustrated in the accompanying drawings anddetailed in the following description. Descriptions of well-knowncomponents and processing techniques are omitted so as to notunnecessarily obscure the embodiments herein. Also, the variousembodiments described herein are not necessarily mutually exclusive, assome embodiments can be combined with one or more other embodiments toform new embodiments. The term “or” as used herein, refers to anon-exclusive or, unless otherwise indicated. The examples used hereinare intended merely to facilitate an understanding of ways in which theembodiments herein can be practiced and to further enable those skilledin the art to practice the embodiments herein. Accordingly, the examplesshould not be construed as limiting the scope of the embodiments herein.

As is traditional in the field, embodiments may be described andillustrated in terms of blocks which carry out a described function orfunctions. These blocks, which may be referred to herein as units ormodules or the like, are physically implemented by analog or digitalcircuits such as logic gates, integrated circuits, microprocessors,microcontrollers, memory circuits, passive electronic components, activeelectronic components, optical components, hardwired circuits, or thelike, and may optionally be driven by firmware. The circuits may, forexample, be embodied in one or more semiconductor chips, or on substratesupports such as printed circuit boards and the like. The circuitsconstituting a block may be implemented by dedicated hardware, or by aprocessor (e.g., one or more programmed microprocessors and associatedcircuitry), or by a combination of dedicated hardware to perform somefunctions of the block and a processor to perform other functions of theblock. Each block of the embodiments may be physically separated intotwo or more interacting and discrete blocks without departing from thescope of the disclosure. Likewise, the blocks of the embodiments may bephysically combined into more complex blocks without departing from thescope of the disclosure.

The accompanying drawings are used to help easily understand varioustechnical features and it should be understood that the embodimentspresented herein are not limited by the accompanying drawings. As such,the disclosure should be construed to extend to any alterations,equivalents and substitutes in addition to those which are particularlyset out in the accompanying drawings. Although the terms first, second,etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are generally onlyused to distinguish one element from another.

Accordingly, embodiments herein disclose a method for auto focus of ascene by an electronic device. The method includes determining, by theelectronic device, at least one region of interest (ROI) in the scenedisplayed in one of a viewfinder and a captured image frame anddetermining, by the electronic device, at least one sub ROI in the atleast one ROI by performing a first level of auto focus on the at leastone ROI using at least one first image sensor of the plurality of imagesensors. Further, the method includes determining, by the electronicdevice, at least one focused sub ROI by performing a second level ofauto focus on the at least one sub ROI using at least one second imagesensor of the plurality of image sensors; and rendering, by theelectronic device, a focus transition for the at least one focused subROI to one of the viewfinder and the captured image frame.

Accordingly, embodiments herein disclose electronic device for autofocus on a scene. The electronic device includes a memory, a processor,a communicator, a plurality of image sensors and a focus optimizationcontroller. The focus optimization controller is configured to determineat least one region of interest (ROI) in the scene displayed in one of aviewfinder and a captured image frame and determine at least one sub ROIin the at least one ROI by performing a first level of auto focus on theat least one ROI using at least one first image sensor of the pluralityof image sensors. The focus optimization controller is also configuredto determine at least one focused sub ROI by performing a second levelof auto focus on the at least one sub ROI using at least one secondimage sensor of the plurality of image sensors, and render a focustransition for the at least one focused sub ROI to one of the viewfinderand the captured image frame.

Conventional methods and systems, recognize objects to be focused andobtain focus statically irrespective of their relevance and importanceto the scene. Unlike to the conventional methods and systems, theproposed method understands the scene and the relevance and importanceof the objects to the scene and allots processing resources accordingly.

Unlike to the conventional methods and systems, the proposed methodoptimizes a number of pixels whose data is processed for phase detectionby finding minimalistic activated pixel distribution incrementally tocross focus quality threshold.

Unlike to the conventional methods and systems, the proposed methodprovides solution for focusing objects that are at edges of a line ofsight by obtaining focal information from other cameras or by intercamera phase detection.

Unlike to the conventional methods and systems, the proposed methodfocuses on sub ROIs based on an importance score assigned and works byselecting multiple cameras. The proposed method uses a combination ofuser behavior and intelligence software for performing dual pixelactivation for focus. Further, the proposed method intelligentlyactivate/deactivate dual pixels in order to detect depth while havingoptimization using multiple cameras. The proposed method uses animportance hierarchy to focus on parts which are deemed to be of moreimportance.

The proposed method provides the following advantages:

1. Power saving: Activation of dual pixel intelligently will decreaseusage of all dual pixels thereby decreasing cpu processing and thereforepower consumption. Processing will be saved by only using boundarypixels for focus and through Gaussian mixture models for focus so as tooptimize the focus delivery and save processing power. The savedprocessing can be used for other purposes.

2. Faster Focus: Due to reduction in processing needed, faster focusingis achieved.

3. User personalization: A hierarchy based focus mechanism will providefocus based on user preference increasing utility for the user. Contentoutput rendering according to the user will relieve user of the effortof creating his preferred content. For example, a gif creation will bedone automatically according to the user, if the user prefers viewinggif content.

4. Edge focus improvement: Current dual pixel auto focus mechanism losesfocus on edges. Proposed solution will use focus information from othercameras in the multi camera apparatus to focus on edges thus solving theinherent problems with dual pixel auto focus.

5. Sub ROI focus: Increase in the overall clarity of the image byfocusing on sub ROIs using the multi camera apparatus as opposed to thestatic focus on the complete image.

FIG. 1 illustrates traditional phase detection autofocus (PDAF),according to the related art.

Conventionally, the Phase detection sensor comes separately withdedicated focus pixels (5% of total pixels) which consist of singlephoto-diode. The PDAF uses phase detection technique on two pixel data(left-looking and right-looking pixels) to obtain phase difference whichwill be minimized. The left and right individual pixels are not arrangedadjacently hence the phase difference being calculated is not veryaccurate. The pixels used for autofocus are static i.e. predefined (5%)and is less accurate and slow. The individual pixels are able to capturelimited image information and hence low light imaging is not good.Accurate focus achievement in all scenarios is not possible due to verylimited number of dedicated pixels being used for auto focus.

Conventionally, when a scene is to be captured by the electronic deviceusing dual pixel phase detection auto-focus sensor, each pixel in a dualpixel array used to capture an image of a scene includes a firstphotodiode and a second photodiode. Further, signals from the firstphotodiode and the second photodiode associated with each pixel in thedual pixel array is sent to a phase difference calculating unit. Basedon the output from the phase difference calculating unit, a focusing andadjustment unit determines auto-focus characteristics for the image.However, the process requires high processing power which in-turnconsumes high battery power. Also, the processing is time-consuming withlow intelligence used, leading to a bad user experience.

Referring now to the drawings, and more particularly to FIGS. 2 through12 , where similar reference characters denote corresponding featuresconsistently throughout the figures, there are shown preferredembodiments.

FIG. 2 illustrates optimized dual pixel autofocus (DPAF), according toan embodiment of the disclosure.

The DPAF includes an imaging sensor which is composed of image pixelswith each pixel containing two photo-dioides (Dual Pixel). The DPAFhandles autofocus. The DPAF uses phase detection technique on singlepixel data (left-photodiode and right-photodiode) to obtain phasedifference which will be minimized. The left and right photodiodes areadjacent and are able to capture the focus info with higher accuracy.The pixels used for autofocus are dynamically selected for each ROIbased on intelligent features like ROI shape, importance, texturedetails, and the like. The individual pixels have two photo sites andhence are capable of handling low light imaging efficiently. Accuratefocus is possible because the number of focus points is higher andintelligently decided for maximum accuracy.

The proposed method for auto-focus selects only strategic pixels whoseleft and right signals are used for focus calculation based onintelligence criteria and multiple cameras.

FIG. 3 illustrates a block diagram of an electronic device for autofocus on a scene, according to an embodiment of the disclosure. Theelectronic device may be, but is not limited to a laptop, a palmtop, adesktop, a mobile phone, a smart phone, Personal Digital Assistant(PDA), a tablet, a wearable device, an Internet of Things (IoT) device,a virtual reality device, a foldable device, a flexible device or animmersive system.

Referring to FIG. 3 , the electronic device 100 may include a memory110, a processor 120, a communicator 130, multiple image sensors 140a-N, an image optimization controller 150 and a display 160.

The memory 110 is configured to store instructions to be executed by theprocessor 120. The memory 110 may include non-volatile storage elements.Examples of such non-volatile storage elements may include magnetic harddiscs, optical discs, floppy discs, flash memories, or forms ofelectrically programmable memories (EPROM) or electrically erasable andprogrammable (EEPROM) memories. In addition, the memory 110 may, in someexamples, be considered a non-transitory storage medium. The term“non-transitory” may indicate that the storage medium is not embodied ina carrier wave or a propagated signal. However, the term“non-transitory” should not be interpreted that the memory 110 isnon-movable. In some examples, the memory 110 can be configured to storelarger amounts of information. In certain examples, a non-transitorystorage medium may store data that can, over time, change (e.g., inRandom Access Memory (RAM) or cache).

The processor 120 communicates with the memory 110, the communicator130, the multiple image sensors 140 a-N, the focus optimizationcontroller 150, and the display 160. The processor 120 is configured toexecute instructions stored in the memory 110 and to perform variousprocesses. The processor may include one or a plurality of processors,may be a general purpose processor, such as a central processing unit(CPU), an application processor (AP), and the like, a graphics-onlyprocessing unit such as a graphics processing unit (GPU), a visualprocessing unit (VPU), and/or an Artificial intelligence (AI) dedicatedprocessor such as a neural processing unit (NPU).

The communicator 130 may include an electronic circuit specific to astandard that enables wired or wireless communication. The communicator130 is configured to communicate internally between internal hardwarecomponents of the electronic device 100 and with external devices viaone or more networks.

The multiple image sensors 140 a-N are configured to receive a scenewhich is displayed in a viewfinder or from a captured image frame. Themultiple image sensors 140 a-N include at least one lens 142 of FIG. 4Athat captures the scene based on the light coming from the scene andfocuses light onto the dual pixel sensor array. Pixels in the firstimage sensor 140 a may include photosensitive elements that convert thelight into digital data and capture the image frame of the scene. Atypical image sensor may, for example, have millions of pixels (e.g.,megapixels) and is configured to capture a series of image frames of thescene based on a single click input from a user. Each of the multipleimage sensors 140 a-N comprises different focal lengths. The image maybe for example still images capturing the scene, video comprisingmultiple images, or a combination thereof.

In an embodiment, the focus optimization controller 150 may include ROIdetection and focus approximation controller 152, sub-ROI segregationand image sensor assignment controller 154, a presentation managementcontroller 156 and a viewfinder controller 158. The focus optimizationcontroller 150 is implemented by processing circuitry such as logicgates, integrated circuits, microprocessors, microcontrollers, memorycircuits, passive electronic components, active electronic components,optical components, hardwired circuits, and the like, and may optionallybe driven by firmware. The circuits may, for example, be embodied in oneor more semiconductors.

The ROI detection and focus approximation controller 152 is configuredto detect multiple ROI in the scene. At least one ROI in the scene isdetected to perform a first level of auto focus. The sub-ROI segregationand image sensor assignment controller 154 is configured to allocate theROI to the first image sensor 140 a of the plurality of image sensors140 a-N to perform the first level of auto focus. Further, the sub-ROIsegregation and image sensor assignment controller 154 is alsoconfigured to perform the first level of auto focus on the at least oneROI using the at least one first image sensor to obtain at least one subROI of the at least one ROI and allocate at least one sub ROI to atleast one second image sensor from the plurality of image sensors toperform a second level of auto focus. Further, the sub-ROI segregationand image sensor assignment controller 154 is configured to perform thesecond level of auto focus on the at least one sub ROI using the atleast one second image sensor to obtain at least one focused sub ROI andrender a focus transition for the at least one focused sub ROI to theviewfinder or the captured image frame.

The presentation management controller 156 is configured to merge autofocused output from the at least one image sensor 140 a-N and aviewfinder controller 158 configured for displaying the auto focusedoutput on the display 160. The auto focused content may be provided in aform based on user preferences such as a video or GIF then the autofocused content is provided as a sequence of the focused image. In casethe user prefers images then a composited completely focused image isprovided. Each of the individual components within the focusoptimization controller 150 is explained in detail from FIG. 4A to FIG.6B.

At least one of the plurality of modules/components of the focusoptimization controller 150 may be implemented through an AI model. Afunction associated with the AI model may be performed through memory110 and the processor 120. The one or a plurality of processors controlsthe processing of the input data in accordance with a predefinedoperating rule or the AI model stored in the non-volatile memory and thevolatile memory. The predefined operating rule or artificialintelligence model is provided through training or learning.

Being provided through learning means that, by applying a learningprocess to a plurality of learning data, a predefined operating rule orAI model of a desired characteristic is produced. The learning may beperformed in a device itself in which AI according to an embodiment isperformed, and/or may be implemented through a separate server/system.

The AI model may consist of a plurality of neural network layers. Eachlayer has a plurality of weight values and performs a layer operationthrough calculation of a previous layer and an operation of a pluralityof weights. Examples of neural networks include, but are not limited to,convolutional neural network (CNN), deep neural network (DNN), recurrentneural network (RNN), restricted Boltzmann Machine (RBM), deep beliefnetwork (DBN), bidirectional recurrent deep neural network (BRDNN),generative adversarial networks (GAN), and deep Q-networks.

The learning process is a method for training a predetermined targetdevice (e.g., a robot) using a plurality of learning data to cause,allow, or control the target device to make a determination orprediction. Examples of learning processes include, but are not limitedto, supervised learning, unsupervised learning, semi-supervisedlearning, or reinforcement learning.

The display 160 is configured to display the auto-focused image on ascreen of the electronic device 100. The display 160 is capable ofreceiving inputs and is made of one of liquid crystal display (LCD),light emitting diode (LED), organic light-emitting diode OLED, and thelike. The display 160 is also configured to display suggestions of acombination of ROIs of the multiple ROIs which the user may want tofocus on.

Although FIG. 3 shows various hardware components of the electronicdevice 100, it is to be understood that other embodiments are notlimited thereon. In other embodiments, the electronic device 100 mayinclude less or more components. Further, the labels or names of thecomponents are used only for illustrative purpose and does not limit thescope of the disclosure. One or more components can be combined togetherto perform same or substantially similar function to rectify the imageby removing the motion characteristic in the image.

FIG. 4A illustrates various functions performed by the ROI detection andfocus approximation controller, according to an embodiment of thedisclosure.

Referring to the FIG. 4A, at operation 401, consider that a lens 142 ofthe image sensor 140 captures the input image which leads to photonsfrom the scene falling on a Dual pixel sensor array 152-a. At operation402, a sensor signal generated by the Dual pixel sensor array 152-a, issent to imaging sensor unit 152-b which generates a digital image atoperation 403. Further, at operation 404 the sub-optimal focused imageand list of ROIs are sent to the sub-ROI segregation and image sensorassignment controller 154.

At operation 405, the digital image is also sent to a phase detectionunit 152-j by the imaging sensor unit 152-b. At operation 406, the phasedetection unit 152-j determines a correction signal and sends thecorrection signal to the lens 142 to capture the image with a correctedphase.

At operation 407, the digital image is sent to an object detection unit152-c which determines the ROI bounding boxes of the objects in theimage. At operation 408, the digital image is also sent to a sceneunderstanding unit 152-d which studies the scene and identifies objectsof relevance in the digital image.

A ROI importance determining unit 152-f receives the ROI bounding boxesof the objects in the image, the objects of relevance in the digitalimage and user preferences of objects to be captured (determined atoperation 409 by an user behavior tracking unit 152-e) as inputs.Further, at operation 410, the ROI importance determining unit 152-fdetermines a ROI importance factor based on the inputs and sends thesame to a camera allocation unit 152-g. The ROI importance determiningunit 152-f determines the relative importance factor of each of the ROIbased on user preferences (typical scenes user might be interested in)and global semantic relevance of each ROI to whole image and generalpreference. In the AI Architecture, a CNN is used for outputtingimportance score for each bounding box.

At operation 411, the camera allocation unit 152-g determines whether acurrent camera is selected. The camera allocation unit 152-g selects thecamera for attaining initial focus based on positions with respect toprimary camera, specifications, line of sight. In AI Architecture:C-class multi class classification neural network, C=no of cameras.

In response to determining that the current camera is selected, aboundary detection and pixel activating unit 152-i at operation 412,sends pixel locations of the pixels to be activated along with anactivation signal to the dual pixel sensor array 152-a. The boundarydetection and pixel activating unit 152-i detects the boundaries of oneor more ROIs and activates them based on pixel density allotted to oneor more ROIs according to the importance factor, for achieving focus.i.e. the pixels which are utilized in phase detection.

In response to determining that the current camera is not selected, atoperation 413 an inter camera communication unit 152-h sends a focalpoint for attaining initial focus to the lens 142. Therefore, unlike tothe conventional methods and systems, the proposed method intelligentlydetermines the objects of importance and actives selected pixels of thedetermined objects of importance. As a result, the electronic device 100saves a large amount of processing resources and power which may beutilized for other applications.

FIG. 4B illustrates a method for determination of ROI importance by anROI importance determining unit, according to an embodiment of thedisclosure.

Referring to the FIG. 4B, the ROI importance determining unit 152-f isconfigured to determine the relative importance factor of each ROI basedon user preferences and global semantic relevance of each ROI to wholeimage and general preference.

Consider that at operation 1, the scene understanding unit 152-ddetermines that the scene comprises “Persons riding bicycle on road” andat operation 2 the Word2vec represents the words in vector forms andfeeds the same to the ROI importance determining unit 152-f. Atoperation 3, the ROI importance determining unit 152-f receives the ROIboundary boxes from the image determined by the object detection unit152-c. At operation 4, the ROI importance determining unit 152-f alsoreceives the user preferences which typical include the scenes that theuser might be interested in form the user behavior tracking unit 152-e.Further, at operation 5, the ROI importance determining unit 152-f whichis a Deep Convolutional Neural Network (DCNN) concatenates the multipleinputs and determines the importance factor for each of the ROI, whichis presented in the table at operation 6. Therefore, from the table atoperation 6, the ROIs enclosing important objects get high importancefactor.

FIG. 4C illustrates functions of a camera allocation unit, according toan embodiment of the disclosure.

Referring to the FIG. 4C, the camera allocation unit 152-g assigns acamera to each ROI based on the camera properties, ROI importance, lineof sight, and the like. At operation 1, the camera allocation unit 152-greceives the importance factor for each of the ROI determined by the ROIimportance determining unit 152-f and at operation 2, receives thevarious camera properties. The camera allocation unit 152-g is a FeedForward Neural Network (FFNN). At operation 3, the camera allocationunit 152-g determines the probability of each of the ROI being assignedto the camera. Further, the camera allocation unit 152-g determines theprobability of each of the ROI being assigned to each of the imagesensors 140 a-N of the electronic device 100 and the camera obtaininghighest probability will be selected.

FIG. 4D illustrates functions of the boundary detection and pixelactivating unit, according to an embodiment of the disclosure.

Referring to the FIG. 4D, the boundary detection and pixel activatingunit 152-i detects the boundaries of one or more ROIs and activates thembased on pixel density allotted to one or more ROIs according to theimportance factor, for achieving focus. i.e. the pixels which areutilized in phase detection.

At operation 1, the boundary detection and pixel activating unit 152-ireceives the bounding boxes of each of the ROI from the cameraallocation unit 152-g. At operation 2, an instance segmentation CNN ofthe boundary detection and pixel activating unit 152-i segments theROIs. At operation 3, an edge detection unit of the boundary detectionand pixel activating unit 152-i detects boundaries of the objects insegmentation mappings. At operation 4, the boundary detection and pixelactivating unit 152-i selects pixels on boundaries based on pixeldensity. More pixels will be selected for boundaries with morecurvature. Once the pixels are selected, at operation 5 the boundarydetection and pixel activating unit 152-i identifies the location ofeach of the selected pixels.

FIG. 4E illustrates a method for controlling a dual pixels array basedon a static pixel usage table, according to the related art.

Referring to the FIG. 4E, at operation 1 a pixel usage lookup table isprovided to an active dual pixel data retriever. The pixel usage lookuptable is a static lookup table with predefined pixels which are used forAF irrespective of the content of the image. Further, the active dualpixel data retriever sends the information of the active dual pixel datafrom the pixel usage lookup table to an aggregate image composing unitwhich is followed by an aggregate image compositing unit. At operation3, the aggregate image compositing unit provides a left image and aright image with the predefined pixels which are used for AF to thephase detection unit. At operation 4, the phase detection unitdetermines a difference in phase between the right image and the leftimage and corrects the lens 142. However, the pixel usage lookup tableis static with predefined pixels and does not take into considerationthe ROI or importance of the ROI in the image. Hence, the staticapproach may miss out on important ROIs in the image and provide focusto un-important contents.

FIG. 4F illustrates a method for dynamically controlling a dual pixelsarray, according to an embodiment of the disclosure.

Referring to the FIG. 4F in conjunction with the FIG. 4E, unlike to theconventional methods and systems for determining the active pixels, inthe proposed method the boundary detection and pixel activating unit152-i intelligently determines the boundary of the objects in the ROIboundary boxes. For example, the ROI boundary box includes the person ona cycle. The boundary detection and pixel activating unit 152-i detectsthe boundary of the person on the cycle and identifies the locations atwhich the pixels needs to be activated (shown with star markings atoperation 1). Further, at operation 2, the activated pixel locations areformulated together in the pixel usage lookup table (operation 3).Therefore, in the proposed method the active pixel locations aredetermined dynamically based on the content in the ROI, shape of thecontent, importance of the content, etc.

FIG. 5A illustrates various functions performed by a sub-ROI segregationand image sensor assignment controller, according to an embodiment ofthe disclosure.

Referring to the FIG. 5A, at operation 501, the sub-optimal focusedimage and list of ROIs received from the ROI detection and focusapproximation controller 152 is sent to a ROI segregation unit 154 a anda landmarks detection unit 154 b. At operation 502, the ROI segregationunit 154 a determines the sub-ROI in the image and shares the same witha pixel density allocating unit 154 e.

At operation 503, the landmarks detection unit 154 b determineslandmarks in the image and shares the same with sub-ROI importancedetermining unit 154 c. At operation 504, the sub-ROI importancedetermining unit 154 c receives the landmarks, the user preferences fromthe user behavior tracking unit 152-e and the landmarks from thesub-ROI. At operation 505, the sub-ROI importance determining unit 154 cdetermines the sub-ROI and the importance factor, and shares the samewith the pixel density allocating unit 154 e. At operation 506, thepixel density allocating unit 154 e also receives the number ofavailable pixels per unit area from the dual pixel sensor configuration154 d. At operation 507, the pixel density allocating unit 154 edetermines sub-ROIs and active pixel density functions and sends it tosub-ROI camera allocation unit 154 f. Operation 508 provides a figurewhere height indicates fraction of pixels activated in that unit area.At operation 509, the sub-ROI camera allocation unit 154 f sends thedata associated with the camera of the multiple cameras, which needs tobe allocated for each of the sub-ROI with active pixel density function.The inter camera communication unit 154 g is configured to communicatebetween the cameras based on the outputs received from the sub-ROIcamera allocation unit 154 f and then send instructions to thepresentation management controller 156.

FIG. 5B illustrates functioning of an ROI segregation unit, according toan embodiment of the disclosure.

Referring to the FIG. 5B, consider an image comprising a person facingcamera upright. Hence, the ROI is the person in the image. The imagewith the ROI determined is provided to the ROI segregation unit 154 awhich is a Deep Convolutional Neural Network. The ROI segregation unit154 a segregates each ROI into sub regions having semanticallyinterpretable components called (Sub-ROIs) by analyzing the ROI image.Therefore, the output of the ROI segregation unit 154 a is that the ROIof the person is segmented into three sub-ROIs which include face of theperson, chest of the person and hands of the person in the image.

FIG. 5C illustrates functioning of a pixel density allocating unit,according to an embodiment of the disclosure.

Referring to the FIG. 5C in conjunction with the FIG. 5A, the pixeldensity allocating unit 154 e receives the input from the ROIsegregation unit 154 a, the landmarks detection unit 154 b and thesub-ROI importance determining unit 154 c. The pixel density allocatingunit 154 e calculates active pixel density function for each sub-ROI asa mixture of Gaussians with each Gaussian centered at one of thelandmarks. Further, the pixel density allocating unit 154 e alsodetermines the variance of each Gaussian intelligently as per therespective landmark's relative importance.

FIG. 6A illustrates various functions performed by a presentationmanagement controller, according to an embodiment of the disclosure.

Referring to the FIG. 6A, at operation 601, the image sensors 140 a-Nreceives the input from the sub-ROI segregation and image sensorassignment controller 154 which indicates specific camera to focus onspecific sub-ROI. Further, the output from each of the image sensors 140a-N is sent to the inter camera communication unit 154 g which sends thesub-ROI list to both a transition effect determining unit 156 a of thepresentation management controller 156 and the lens 142 in operation602. Further, at operation 603, the lens 142 captures the sub-ROI and atoperation 604, the dual pixel sensor array 152-a sends the sensor datato the imaging sensor unit 152-b. At operation 605, the transitioneffect determining unit 156 a also receives the user preferences anddetermines the effects which may needs to be applied while transitionthe content to be rendered. At operation 606, the focused content issent to an output rendering unit 156 b which displays the auto-focusedcontent on the display 160 of the electronic device 100.

FIG. 6B illustrates various functions performed by a transition effectdetermining unit, according to an embodiment of the disclosure.

Referring to the FIG. 6B, the transition effect determining unit 156 ais a deep convolutional neural network. The transition effectdetermining unit 156 a receives the output from the inter cameracommunication unit 154 g and the user preferences from the user behaviortracking unit 152-e. The user preference includes instant focus, orderedfocus and animation.

Both the input features are concatenated and the instant focus isdetermined and applied to the final image. Then the focused image isrendered on the output rendering unit 156 b.

FIG. 7A to 7F are flow diagrams illustrating a sequence of operationsfor two levels of auto focus with respect to an electronic device,according to various embodiments of the disclosure. FIG. 7A illustratesthe overall method flow.

Referring to FIG. 7 , at operation 702, the method includes theelectronic device 100 detecting the ROI in the scene displayed in theviewfinder or from a captured image frame. For example, in theelectronic device 100 described in the FIG. 3 , the focus optimizationcontroller 150 is configured to detect the ROI in the scene displayed inthe viewfinder or from a captured image frame.

At operation 704, the method includes the electronic device 100allocating the ROI to the first image sensor 140 a from the plurality ofimage sensors 140 a-N to perform the first level of the auto focus. Forexample, in the electronic device 100 described in the FIG. 3 , thefocus optimization controller 150 is configured to allocate the ROI tothe first image sensor 140 a from the plurality of image sensors 140 a-Nto perform the first level of the auto focus.

At operation 706, the method includes the electronic device 100performing the first level of auto focus on the at least one RoI usingthe first image sensor 140 a to obtain the at least one sub RoI of atleast one Rot For example, in the electronic device 100 described in theFIG. 3 , the focus optimization controller 150 is configured to performthe first level of auto focus on the at least one RoI using the firstimage sensor 140 a to obtain the at least one sub RoI of the RoIs.

At operation 708, the method includes the electronic device 100allocating the at least one sub RoI to the at least one second imagesensor 140 b from the plurality of image sensors 140 a-N to perform thesecond level of the auto focus. For example, in the electronic device100 described in the FIG. 3 , the focus optimization controller 150 isconfigured to allocate the at least one sub RoI to the at least onesecond image sensor 140 b from the plurality of image sensors 140 a-N toperform the second level of the auto focus.

At operation 710, the method includes the electronic device 100performing the second level of auto focus on the at least one sub RoIusing the at least one second image sensor 140 b to obtain the at leastone focused sub RoI. For example, in the electronic device 100 describedin the FIG. 3 , the focus optimization controller 150 is configured toperform the second level of auto focus on the at least one sub RoI usingthe at least one second image sensor 140 b to obtain the at least onefocused sub RoI.

At operation 712, the method includes the electronic device 100rendering the focus transition for the at least one focused sub RoI tothe viewfinder or the captured image frame. For example, in theelectronic device 100 described in the FIG. 3 , the focus optimizationcontroller 150 is configured to render the focus transition for the atleast one focused sub RoI to the viewfinder or the captured image frame.

The various actions, acts, blocks, operations, or the like in the method700 may be performed in the order presented, in a different order orsimultaneously. Further, in some embodiments, some of the actions, acts,blocks, operations, or the like may be omitted, added, modified,skipped, or the like without departing from the scope of the disclosure.

FIG. 7B further illustrates the operation 704. At operation 704A, theplurality of parameters associated with the at least one ROI isdetermined, wherein the plurality of parameters comprises a focal lengthof an image sensor, a type of the at least one ROI, an importance of theat least one ROI, a line of sight of each image sensor of the electronicdevice 100, and a resolution of each image sensor of the electronicdevice 100. At operation 704B, the at least one first image sensor 140 afrom the plurality of image sensors 140 a-N is selected to perform thefirst level of auto focus based on the plurality of parameters. Atoperation 704C, the at least one ROI to the at least one first imagesensor from the plurality of image sensors 140 a-N is allocated toperform the first level of auto focus.

Based on the image sensor allotted, boundaries for each of the ROI s arefurther detected and appropriately the corresponding dual pixels in thedual pixel array are activated. Phase detection auto focus is performedon the selected camera to get a sub-optimal focused image at operation706 of FIG. 7A. The sub-optimal focused image is further sent to the ROIsegregation unit 154 a and pixel density allocating unit 154 e. The ROIis segregated into a plurality of sub-ROIs. At least one landmark isdetected on each of the plurality of the sub-ROIs. An importance factoris estimated by a sub-ROI importance determining unit based on userpreferences and reference images similar to the landmark detected. Thisis further illustrated in FIG. 7C. At operation 706A, boundary pixels ofthe ROI are detected for auto focus. A corresponding dual pixelphoto-diode set in the dual pixel array is activated at operation 706B.The first level of auto focus is performed corresponding to the detectedboundary pixels of the at least one ROI using the activated diode toobtain the sub-ROI of the at least one ROI.

At operation 708 of FIG. 7A, a pixel density allocating unit is used toactivate the corresponding and appropriate dual pixel sensors from thedual pixel array and subsequently allocates a camera among the pluralityof cameras on the electronic device 100 such that the camera supportsthe pixel density function obtained from the activated dual pixels (orthe set of two photo-diodes of the dual pixels). In an embodiment, thepixel density function is appropriated by the sub-ROI using GaussianMixture model based on the detected landmarks. This is furtherillustrated in FIG. 7D. At operation 708A, the at least one ROI isfurther segregated into a plurality of sub-ROIs.

Further operations may include detecting at least one landmark on eachof the plurality of sub-ROIs (operation 708A), detecting at least onelandmark on each of the plurality of sub-ROIs (operation 708B),estimating an importance factor for each sub-ROI based on the detectedat least one landmark on each of the plurality of sub-ROIs (operation708C), selecting the at least one sub-ROI from the plurality of sub-ROIsbased on the importance factor and the detected landmarks (operation708D), selecting the at least one second image sensor supporting dualpixel density function from the plurality of image sensors 140 a-N, andallocating the at least one sub-ROI to the at least one selected secondimage sensor for performing the second level of auto focus (Operations708E and 708F).

Using the camera allotted and the activated dual pixel sensors, thesecond level of autofocus corresponding to the detected boundary pixelsof the sub ROI using the activated photo-diodes to obtain a focusedsub-ROI at operation 710 of FIG. 7A. As FIG. 7E illustrates, theoperations for performing the second level of auto focus includedetermining the pixel density function (operation 710A), activating theset of two photo-diodes of the at least one second image sensorcorresponding to the sub-ROI for performing the second level of autofocus based on the determined pixel density function (operation 710B),and performing the second level of auto focus corresponding to thedetected boundary pixels of the at least one sub ROI using the at leastone activated diode to obtain the at least one focused sub-ROI(operation 710C).

Subsequently the focused sub-ROI is merged with the input scene by thepresentation management controller 156 at operation 712 of FIG. 7A. Atransition effect determining unit is used to merge the focused sub-ROIsbased on user preferences and the importance factors and subsequentlyprovided to the viewfinder or the camera lens of the electronic deviceready to be captured as a stored image. As illustrated in FIG. 7F,operations include merging the at least one focused sub-RoI obtainedfrom the at least one second image senor to obtain an optimal sub-RoI(operation 712A), determining the focus transition for displaying theoptimal sub-RoI (operation 712B), and rendering the focus transition forthe optimal sub-RoI to the viewfinder or the captured image (operation712C).

FIGS. 8A and 8B are examples illustrating multiple ROIs with different Zorders being captured by an electronic device, according to anembodiment of the disclosure.

Referring to FIGS. 8A and 8B, at operation 802, consider an imagecomprising two related objects such as for example two slices of pizzawhere one is placed on a table and the other is held by a user. Theslide held by the user is at an edge of the image and hence not focusedwhen the image is captured using existing methods and systems. Atoperation 804, consider another scenario where the object in thebackground (slice of the pizza placed on the table) is related to theobject ahead i.e., the slice held by the user. Yet the slice in thebackground is not focused when the image is captured using existingmethods and systems.

Consider the image is captured using the proposed method. At operation806, the electronic device 100 uses the first image sensor 140 a and thesecond image sensor 140 b for intelligent dual pixel capture of both theslices of the pizza in the image. The electronic device 100intelligently determines the ROI in the image as the region comprisingboth the slices of the pizza. Further, the electronic device 100 alsodetermines the sub-ROIs as the slice of the pizza in the background(operation 808) and the slice of the pizza held by the user (operation810). Then the electronic device 100 allocates pixel density functionfor each of the sub ROIs based on the landmarks in the image.

Further, the sub-ROI comprising the pizza slice in the background isallocated to the first image sensor 140 a and the sub-ROI comprising theslice of the pizza held by the user is allocated to the second imagesensor 140 b. At operation 812, the electronic device 100 merges theoutput from the first image sensor 140 a and the second image sensor 140b. The electronic device 100 also determines the transition effect to beapplied for the final image and displays the final image with focus onboth the objects in the image on the screen of the electronic device100.

FIG. 9 is an example illustrating a comparison of capturing of contentin the edge of the images by an electronic device, according to anembodiment of the disclosure.

Referring to the FIG. 9 , at operation 902, the image is capturing usingthe existing dual pixel phase detection and auto focus capture where allthe dual pixels are considered. The focus on the edge pixels is notefficient due less amount of the light falling on the photodiodes on theedges.

At operation 904, the electronic device 100 captures the image by usingthe proposed method which selectively activates the dual pixels in thedual pixel array only in the ROI from the main camera. If the image isnot focused, then another processing operation is applied in which thedual pixels of the secondary camera which has a bigger depth of view isused. Therefore, unlike to the conventional methods and systems, in theproposed method the edges can be focused using the data from thesecondary camera.

FIG. 10 is an example illustrating the capture of a sudden intrudingobject by an electronic device, according to an embodiment of thedisclosure.

Consider an example scenario where the electronic device 100 is focusingon a specific road for capturing the image and suddenly an intruder suchas a bird appears in front of the electronic device 100, blocking theview of the road. In the conventional method and system of capturing theimage, the focus of the camera of the electronic device 100 shifts fromthe road to the bird as shown in operation 1004. A similar scenario maybe intentionally used by unlawful entities to perform unlawfulactivities such as theft by using an object which shifts the focus fromthe background to the closer object.

In the proposed method, at operation 1006, the electronic device 100determines the ROIs in the image along with the importance of the ROIsdetected. At operation 1008, the electronic device 100 identifies thefirst ROI is the road, the importance factor is 0.89 and therefore, theallotted camera is the first image sensor 140 a in the sub-optimalfocused image. Similarly, the second ROI is the owl, the importancefactor is 0.75 and therefore, the allotted camera is the second imagesensor 140 b. Further, at operation 1010, the electronic device 100determines the sub ROIs in the image as the road, an Owl face and an Owlbody. Also, the electronic device 100 identifies the first sub-ROI isthe road, the importance factor is 0.89 and the allotted camera is thefirst image sensor 140 a; the second sub-ROI is the Owl Face, theimportance is 0.86 and the allotted camera is the second image sensor140 b and the third sub-ROI is the Owl body, the importance is 0.65 andthe allotted camera is the third image sensor 140 c. The proposed methodincludes the electronic device 100 intelligently performing the cameraallotment and selectively activating the dual pixels in the dual pixelarray to focus on the objects in the image. At operation 1012, theelectronic device 100 merges the optimally focused image with the inputscene using the presentation management controller 156 to obtain thefinal image where both the focus is on both the road and the owl.Therefore, unlike to the conventional methods and systems where thefocus shifts when the intruder is introduced into the scene, in theproposed method the electronic device 100 focuses intelligently on boththe intruder and the object being previously focused due to lesserprocessing.

FIG. 11A illustrates a comparison of existing method and proposed methodof focusing on the objects on the viewfinder, according to an embodimentof the disclosure.

Referring to the FIG. 11A, at operation 1102 illustrates theconventional methods of capturing the image in the simple focus whenthere are multiple objects (i.e., multiple ROIs) in the viewfinder. Forexample, in the image provided there are two persons, two dogs and adoll. Since there are multiple objects the user may want to focus onspecific objects and ignore the other objects in the scene. However, thefocus mechanism does not provide the options to the user.

Unlike to the conventional methods and systems, in operation 1104 theelectronic device 100 intelligently determines the ROIs, the sub-ROIs,the corresponding importance of the sub-ROIs and suggests the possiblecombinations of the objects to which the user may want to focus in thescene. For example, in the above example case the electronic device 100provides suggestion to the user to select from such as 1) Focus on thePersons only, 2) Focus on the persons and the dogs and 3) Focus on thepersons, the dogs and the doll in the image. The user can select one ofthe options and the user preferred focus is applied while capturing theimage. The suggestions may be provided as the proposed method is lowtime consuming and low processing resources are required for the same.

FIG. 11B is an example illustrating the operation-by-operation procedurefor providing the real-time suggestions for the objects to be focused onthe viewfinder, according to an embodiment of the disclosure.

Referring to the FIG. 11B, in conjunction to the FIG. 11 , at operation1106, the electronic device 100 detects the multiple ROIs in the scenedisplayed in the viewfinder and at operation 1108 determines theimportance of each of the multiple ROIs, allocates the cameras for eachof the ROIs in the scene. In the scene provided the electronic device100 detects the first ROI as the person 1 with importance 0.92 andallocates the first image sensor 140 a to capture the same. The secondROI is the dog 1 with importance 0.74 and allocates the first imagesensor 140 a to capture the dog 1. The third ROI is the person 2 withimportance 0.89 and allocates the second image sensor 140 b to capturethe person 2 and the fourth ROI is the dog 2 with importance 0.71 andallocates the second image sensor 140 b to capture the dog 2. Further,the electronic device 100 distributes the activated pixels on the ROIboundaries in operation 1110.

Further, at operation 1108 the electronic device 100 determines thesub-ROIs, further allocates optimum cameras of the multiple cameras tofocus on the sub-ROIs and activated pixels are distributed based onestimated Gaussian mixture (with mean at landmarks). Further, atoperation 1112, the electronic device 100 intelligently providessuggestions based on multiple combinations of the ROIs in the scene forappropriate auto focus before capturing the image or video of the inputscene and the user can select in real-time in the viewfinder.

FIG. 12 is an example illustrating a scenario of threat identificationin the scene by the electronic device 100, according to an embodiment ofthe disclosure.

Referring to FIG. 12 , in the related art, auto focus systems fail torecognize threats in distance as illustrated in operation 1202. Inaccordance with the disclosure, tiger, surrounding environment and thethreat are consistently focused. ROIs with global importance like crime,havoc etc. are given a high importance factor helping in faster threatidentification in operation 1204. As illustrated operation 1206, theelectronic device 100 determines the multiple ROIs in the input scenewhich are the tiger and the threat (hunter and hunting dog) using theimage sensors 140 a-N and the ROI detection and focus approximationcontroller 152. At operation 1208, the sub-ROI segregation and imagesensor assignment controller 154 determines the importance factors andthe sub-ROIs. The hunter and hunting dog are given higher importance.Accordingly optimum cameras are allotted to focus on the sub-ROI and thecorresponding dual pixels activated. At operation 1210, an optimallyfocused image is obtained and the optimally focused image is merged withthe input scene by the electronic device 100.

In another example, current dual pixel autofocus system faces problem totrack objects with very fast paced motion due to higher processing in agrid and so loses focus on fast paced objects. The proposed methodimproves autofocusing speed due to low computation while offeringaccurate focus and greatly improves the seamless focus tracking in fastpaced scenes.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. A method for auto focus of a scene by anelectronic device comprising a plurality of image sensors, the methodcomprising: determining, by the electronic device, at least one regionof interest (ROI) in the scene displayed in one of a viewfinder and acaptured image frame; determining, by the electronic device, at leastone sub-ROI in the at least one ROI by performing a first level of autofocus on the at least one ROI using at least one first image sensor ofthe plurality of image sensors; determining, by the electronic device,at least one focused sub-ROI by performing a second level of auto focuson the at least one sub-ROI using at least one second image sensor ofthe plurality of image sensors; and rendering, by the electronic device,a focus transition for the at least one focused sub-ROI to one of theviewfinder and the captured image frame.
 2. The method of claim 1,wherein the determining of the at least one sub-ROI in the at least oneROI by performing the first level of auto focus on the at least one ROIusing the at least one first image sensor of the plurality of imagesensors comprises: determining, by the electronic device, a relativeimportance factor for each of the at least one ROI; allocating, by theelectronic device, the at least one ROI to the at least one first imagesensor based on the relative importance factor and a plurality ofparameters; selecting, by the electronic device, dual pixels using apixel density function to focus on of the at least one ROI for focuscalculation; activating, by the electronic device, a set of twophoto-diodes of the at least one first image sensor corresponding to theselected dual pixels for performing the first level of auto focus;performing, by the electronic device, the first level of auto focuscorresponding to the selected dual pixels of the at least one ROI usingan activated diode; and determining, by the electronic device, the atleast one sub-ROI in the at least one ROI based on the first level ofauto focus performed on the at least one ROI.
 3. The method of claim 2,wherein the selected dual pixels are located at a boundary of the atleast one ROI.
 4. The method of claim 2, wherein the determining of therelative importance factor for each of the at least one ROI comprises:determining, by the electronic device, a semantic relevance of the atleast one ROI to the scene; determining, by the electronic device, atleast one object in the at least one ROI and a user preference withrespect to the determined at least one object; concatenating, thesemantic relevance of the at least one ROI, the at least one object inthe at least one ROI and the user preference with respect to thedetermined at least one object; and determining, by the electronicdevice, the relative importance factor for each of the at least one ROIbased on the concatenation.
 5. The method of claim 2, wherein theallocating of the at least one ROI to the at least one first imagesensor based on the relative importance factor and the plurality ofparameters comprises: determining, by the electronic device, theplurality of parameters associated with the at least one ROI, theplurality of parameters comprising a focal length of an image sensor, atype of the at least one ROI, the importance of the at least one ROI, aline of sight of each image sensor of the electronic device, and aresolution of each image sensor of the electronic device; selecting, bythe electronic device, the at least one first image sensor from theplurality of image sensors to perform the first level of auto focusbased on the plurality of parameters; and allocating, by the electronicdevice, the at least one ROI to the at least one first image sensor fromthe plurality of image sensors to perform the first level of auto focus.6. The method of claim 1, wherein the determining of the at least onefocused sub-ROI by performing the second level of auto focus on the atleast one sub-ROI using the at least one second image sensor of theplurality of image sensors comprises: allocating, by the electronicdevice, the at least one sub-ROI to at least one second image sensorfrom the plurality of image sensors to perform the second level of autofocus; selecting, by the electronic device, dual pixels using a pixeldensity function to focus on the at least one sub-ROI for focuscalculation; activating, by the electronic device, a set of twophoto-diodes of the at least one second image sensor corresponding tothe selected dual pixels for performing the second level of auto focus;and performing, by the electronic device, the second level of auto focuscorresponding to the selected dual pixels of the at least one sub-ROIusing at least one activated pixel to obtain the at least one focusedsub-ROI.
 7. The method of claim 6, wherein the pixel density function isdetermined as an active pixel density function for the at least onefocused sub-ROI as a mixture of Gaussians, and wherein with eachGaussian is centered at a landmark.
 8. The method of claim 6, whereinthe allocating of the at least one sub-ROI to at least one second imagesensor from the plurality of image sensors to perform the second levelof auto focus comprises: segregating, by the electronic device, the atleast one ROI into a plurality of sub-ROIs; detecting, by the electronicdevice, at least one landmark on each of the plurality of sub-ROIs;estimating, by the electronic device, an importance factor for eachsub-ROI based on the detected at least one landmark on each of theplurality of sub-ROIs; selecting, by the electronic device, the at leastone sub-ROI from the plurality of sub-ROIs based on the importancefactor and the detected at least one landmark; selecting, by theelectronic device, the at least one second image sensor supporting dualpixel density function from the plurality of image sensors; andallocating, by the electronic device, the at least one sub-ROI to the atleast one selected second image sensor for performing the second levelof auto focus.
 9. The method of claim 8, wherein the estimating of theimportance factor for each sub-ROI based on the detected at least onelandmark on each of the plurality of sub-ROIs comprises: determining, bythe electronic device, a number of ROIs detected in the sub-ROI;determining, by the electronic device, a user preference correspondingto the sub-ROI; and estimating, by the electronic device, the importancefactor for the sub-ROI using the number of ROIs detected in the sub-ROIand the user preference corresponding to the sub-ROI.
 10. The method ofclaim 1, wherein the rendering of the focus transition for the at leastone focused sub-ROI to one of the viewfinder and the captured imageframe comprises: merging, by the electronic device, the at least onefocused sub-ROI obtained from the at least one second image sensor toobtain an optimal sub-ROI; determining, by the electronic device, thefocus transition for displaying the optimal sub-ROI; and rendering, bythe electronic device, the focus transition for the optimal sub-ROI toone of the viewfinder and the captured image.
 11. The method of claim 9,wherein the focus transition is determined based on at least one of auser preference and a hierarchy important of the at least one ROI, atype of the at least one ROI, or an importance of the at least one ROI.12. An electronic device for auto focus of a scene, wherein theelectronic device comprises: a plurality of image sensors; a memory; aprocessor coupled to the memory; a communicator coupled to the memoryand the processor; and a focus optimization controller coupled to thememory, the processor and the communicator, wherein the focusoptimization controller configured to: determine at least one region ofinterest (ROI) in the scene displayed in one of a viewfinder and acaptured image frame, determine at least one sub-ROI in the at least oneROI by performing a first level of auto focus on the at least one ROIusing at least one first image sensor of the plurality of image sensors,determine at least one focused sub-ROI by performing a second level ofauto focus on the at least one sub-ROI using at least one second imagesensor of the plurality of image sensors, and render a focus transitionfor the at least one focused sub-ROI to one of the viewfinder and thecaptured image frame.
 13. The electronic device of claim 12, wherein thefocus optimization controller is further configured, when determiningthe at least one sub-ROI in the at least one ROI by performing the firstlevel of auto focus on the at least one ROI using the at least one firstimage sensor of the plurality of image sensors, to: determine a relativeimportance factor for each of the at least one ROI; allocate the atleast one ROI to the at least one first image sensor based on therelative importance factor and a plurality of parameters; select dualpixels using a pixel density function to focus on of the at least oneROI for focus calculation; activate a set of two photo-diodes of the atleast one first image sensor corresponding to the selected dual pixelsfor performing the first level of auto focus; perform the first level ofauto focus corresponding to the selected dual pixels of the at least oneROI using an activated diode; and determine the at least one sub-ROI inthe at least one ROI based on the first level of auto focus performed onthe at least one ROI.
 14. The electronic device of claim 13, wherein theselected dual pixels are located at a boundary of the at least one ROI.15. The electronic of claim 13, wherein the focus optimizationcontroller is further configured, when determining the relativeimportance factor for each of the at least one ROI, to: determine asemantic relevance of the at least one ROI to the scene; determine atleast one object in the at least one ROI and a user preference withrespect to the determined at least one object; concatenate the semanticrelevance of the at least one ROI, the at least one object in the atleast one ROI and the user preference with respect to the determined atleast one object; and determine the relative importance factor for eachof the at least one ROI based on the concatenation.
 16. The electronicdevice of claim 13, wherein the focus optimization controller is furtherconfigured, when allocating the at least one ROI to the at least onefirst image sensor based on the relative importance factor and theplurality of parameters, to: determine the plurality of parametersassociated with the at least one ROI, wherein the plurality ofparameters comprises a focal length of an image sensor, a type of the atleast one ROI, the importance of the at least one ROI, a line of sightof each image sensor of the electronic device, and a resolution of eachimage sensor of the electronic device; select the at least one firstimage sensor from the plurality of image sensors to perform the firstlevel of auto focus based on the plurality of parameters; and allocatethe at least one ROI to the at least one first image sensor from theplurality of image sensors to perform the first level of auto focus. 17.The electronic device of claim 12, wherein the focus optimizationcontroller is further configured, when determining the at least onefocused sub-ROI by performing the second level of auto focus on the atleast one sub-ROI using the at least one second image sensor of theplurality of image sensors comprises, to: allocate the at least onesub-ROI to at least one second image sensor from the plurality of imagesensors to perform the second level of auto focus; select dual pixelsusing a pixel density function to focus on the at least one sub-ROI forfocus calculation; activate a set of two photo-diodes of the at leastone second image sensor corresponding to the selected dual pixels forperforming the second level of auto focus; and perform the second levelof auto focus corresponding to the selected dual pixels of the at leastone sub-ROI using at least one activated pixel to obtain the at leastone focused sub-ROI.
 18. The electronic device of claim 17, wherein thepixel density function is determined as an active pixel density functionfor the at least one focused sub-ROI as a mixture of Gaussians, andwherein with each Gaussian is centered at a landmark.
 19. The electronicdevice of claim 17, wherein the focus optimization controller is furtherconfigured, when allocating the at least one sub-ROI to at least onesecond image sensor from the plurality of image sensors to perform thesecond level of auto focus, to: segregate the at least one ROI into aplurality of sub-ROIs; detect at least one landmark on each of theplurality of sub-ROIs; estimate an importance factor for each sub-ROIbased on the detected at least one landmark on each of the plurality ofsub-ROIs; select the at least one sub-ROI from the plurality of sub-ROIsbased on the importance factor and the detected at least one landmark;select the at least one second image sensor supporting dual pixeldensity function from the plurality of image sensors; and allocate theat least one sub-ROI to the at least one selected second image sensorfor performing the second level of auto focus.
 20. A non-transitorycomputer-readable recording medium having recorded thereon a program forexecuting, the method of claim 1.